The Internet continues to grow without bounds. Networks have become indispensable for conducting all forms of business and personal communications. Networked systems allow one to access needed information rapidly, collaborate with partners, and conduct electronic commerce. The benefits offered by Internet technologies are too great to ignore. However, as with all technology advances, a trade-off ensues. While computer networks revolutionize the way one does business, the risks introduced can be substantial. Attacks on networks can lead to lost money, time, reputation, and confidential information.
One primary danger to avoid is having outside intruders gaining control of a host on a network. Once control is achieved, private company files can be downloaded, the controlled host can be used to attack other computers inside the firewall, or the controlled host can scan or attack computers anywhere in the world. Many organizations have pursued protecting their borders by the implementation of firewalls, Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS). Studies from the FBI and other security organizations have determined that the majority of network security incidents are initiated inside the firewall. As a result, monitoring the inside of the network is becoming more important with time.
Firewalls merely limit access between networks. Firewalls are typically designed to filter network traffic based on attributes such as source or destination addresses, port numbers, or transport layer protocols. Firewalls are susceptible to maliciously crafted traffic designed bypass the blocking rules established. Additionally, almost all commercially available IDS are signature based detection systems or anomaly based systems.
Signature based detection systems piece together the packets in a connection to collect a stream of bytes being transmitted. The stream is then analyzed for certain strings of characters in the data commonly referred to as “signatures.” These signatures are particular strings that have been discovered in known exploits. The more signatures that are stored in a database, the longer it takes to do on exhaustive search on each data stream. For larger networks with massive amounts of data transferred, a string comparison approach is unfeasible. Substantial computing resources are needed to analyze all of the communication traffic.
Besides, even if a known exploit signature has been discovered, the signature is not useful until it is has been installed and is available to the network. In addition, signature analysis only protects a system from known attacks. Yet, new attacks are being implemented all the time. Unfortunately, a signature based detection system would not detect these new attacks and leave the network vulnerable.
Another approach to intrusion detection includes detection of unusual deviation from normal data traffic commonly referred to as “anomalies.” Like signature-based detection systems, many current anomaly based intrusion detection systems only detect known methods of attacks. Some of these known anomaly based attacks include TCP/IP stack fingerprinting, half-open attacks, and port scanning. However, systems relying on known attacks are easy to circumnavigate and leave the system vulnerable. In addition, some abnormal network traffic happens routinely, often non-maliciously, in normal network traffic. For example, an incorrectly entered address could be sent to an unauthorized port and be interpreted as an abnormality. Consequently, known anomaly based systems tend to generate an undesirable number of false alarms, which creates a tendency to have all alarms generated to become ignored.
Some known intrusion detection systems have tried to detect statistical anomalies. The approach is to measure a baseline and then trigger an alarm when deviation is detected. For example, if a system typically has no traffic from individual workstations at 2 am, activity during this time frame would be considered suspicious. However, baseline systems have typically been ineffective because the small amount of malicious activity is masked by the large amounts of highly variable normal activity. On the aggregate, it is extremely difficult to detect the potential attacks.
Other intrusion detection systems compare long term profiled data streams to short term profiled data streams. One such system is described in U.S. Pat. No. 6,321,338 to Porras et al. entitled “Network Surveillance.” The system described in this patent does not necessarily analyze all the network traffic, but instead focus on narrow data streams. The system filters data packet into various data streams and compares short-term profiles to profiles collected over a long period. However, data traffic is typically too varied to meaningfully compare short-term profiles to long-term profiles. For example, merely because the average FTP streams may be 3 megabytes over the long term does not indicate that a 20-megabyte stream is an anomaly. Consequently, these systems generate a significant amount of false alarms or the malicious activity can be masked by not analyzing the proper data streams.
Consequently, a scalable intrusion detection system that effectively tracks characterized and tracks network activity to differentiate abnormal behavior. Due to the impracticality of analyzing all the data flowing through the network, the system cannot rely on signature-based methods. The detection system must be able to function even with the data traffic of larger networks. In addition, the system needs to quickly and efficiently determine if the network has undergone an attack without an excessive amount of false alarms.
Flow based intrusion detection systems have been introduced over the past few years and have evolved into what is commonly referred to as “behavior based” intrusion detection systems. This type of intrusion detection system commonly analyses traffic patterns by assembling the packets into traffic sessions, or flows, and then monitors changes in flow based characteristics. A system of this type is described in detail in Lancope patent application Ser. No. 10/000,396.
A characteristic of flow-based or behavior IDS is that this class of system provides much more context as relates to the network traffic than a signature-based system. A signature-based system is typically limited to processing the packets passing through the device at any given time while looking for explicit patterns in the packet data. In the instance that a pattern is matched, an event is triggered, thus notifying the user that a possible intrusion is taking place.
A flow-based analysis system maintains a record of communications for each host and updates it's knowledge of the host over time. For example, a given host may commonly operate as a client on port 80 (HTTP), this behavior will be assimilated and maintained by the flow-based system within a profile that is dedicated to the specific host. If the host suddenly begins operating as an HTTP server, this action triggers an alert from the flow-based system that the behavior of the host has changed. The creation over time of a “host profile” and the deviation in performance from this profile is a common characteristic of a flow-based behavior IDS.
There are two common embodiments of this type of system. In one case, the packets are captured directly by the IDS and the packet headers are processed to assemble the information into flows. While this implementation provides the most detailed set of data and hence the results, the difficulty in today's high-speed networks of getting access to the data wherever it is needed limits the usefulness to the “network cores” and protection of other high value assets. Further, the Installation of a packet capture IDS system at each remote location is often not feasible system configuration option.
To overcome this limitation, a second embodiment is based on what is commonly referred to as Netflow data, wherein a router provides the Netflow data. A Netflow record contains information about each flow that passes through the router and provides a digest of the communications showing hosts that were involved, services that were used, and how much data was exchanged. Through use of the router provided information, Netflow overcomes the limitations of requiring a separate packet capture IDS be installed at each location on the network. However, Netflow does not contain any packet level information, which limits the amount of detail available about the communications session. A possible source of data for network analysis is known as sFlow. As described in RFC-3176, sFlow is a technology designed for network monitoring based on packet sampling that is derived from work performed at the University of Geneva and CERN in 1991. sFlow operates by randomly sampling one out of every “n” data packets at a switch or router. The packet header along with forwarding tables and interface counters as well as additional packet information are delivered to an sFlow agent residing on the switch, wherein the sFlow agent forwards the captured information to an sFlow collector for processing. This technology is highly scalable, due to the fact that sample rates can be reduced on higher speed networks without causing a reduction in accuracy of traffic measurement.
A possible source of data for network analysis is known as sFlow. As described in RFC-3176, sFlow is a technology designed for network monitoring based on packet sampling that is derived from work performed at the University of Geneva and CERN in 1991. sFlow operates by randomly sampling one out of every “n” data packets at a switch or router. The packet header along with forwarding tables and interface counters as well as additional packet information are delivered to an sFlow agent residing on the switch, wherein the sFlow agent forwards the captured information to an sFlow collector for processing. This technology is highly scalable, due to the fact that sample rates can be reduced on higher speed networks without causing a reduction in accuracy of traffic measurement.
The basic theory of packet sampling comprises the aspect of obtaining a significantly large number of samples over a period of time in order to minimize the sampling error for a specified class of traffic. The error rate for sampling is approximated by an equation showing that the error rate equals 196 times the square root of (1/N), where “N” is the number of observed samples. For example, given approximately 10,000 samples of a given class, there is a predicted error rate of less than 2% for the traffic reported.
Typically, pattern-matching systems (e.g., signature-based systems) are limited in their ability to deal with sampled data that is provided by sFlow for several reasons. First, many of the common signatures require several hundred bytes (or more) of the packet data in order to detect an event. By default, sFlow only returns the first 128 bytes of a packet header. Secondly, data streams cannot be easily assembled (concatenating several packets together) for the reason that it is highly unlikely that all of the packets will be selected for sampling.
This particular aspect essentially limits signature-based systems operating with sFlow to those signatures that occur in the first 128 bytes of the first packet containing payload in a flow, and only in those flows in which the first packet with payload is sampled. Third, the “event detection” mechanism used by signature-based systems does not allow long-term learning to add to the value. These are severe limitations on the capabilities of signature-based systems operating with sFlow. To overcome these limitations, a system is needed that provides additional context, as is described in the following disclosure.